Collecting, sorting, and reporting large volumes of data have become increasingly important as valuable data continues to grow at significant rates. Originally, businesses and government agencies hired individuals to manage large collections of data, which included storing and indexing files in large record centers. As the cost of managing the physical files increased, investing in a less expensive, more efficient mechanical solution became more worthwhile.
After the introduction of the computer, a number of advanced techniques emerged to provide automated data management. Database models were developed allowing information to be conceptualized, structured, and manipulated without hardware-specific dependency limitations. Navigational, hierarchical, network and relational database models provided rapid access to large amounts of data through the use of computer applications. Such database models often use techniques such as data mining, data warehousing, and data marts for effective data management. The relational database model has become the most prevalent database model in use today, because it provides data independence from hardware and store implementation, while providing an automatic navigation (or a high level, nonprocedural language) for accessing data.
Since the introduction of databases, the size of databases has grown from a few megabytes of data for applications just a few years ago to several terabytes of data for today's applications, such as mailing lists, customer information for retail businesses, and the like. As the amount of data increases, the cost of storage space and data management also increases. Current database servers use a common technique of locating data by utilizing index files. The index files cross-reference sub-sets of information with a physical location of the data within the database. Even database servers based on hash algorithms must rely on index files, particularly when the data is to be accessed by more than one element. When creating a database table in a relational database, the designer must understand how the table will be used and create appropriate indexes. Unfortunately, when the nature of the data request does not match the indexing of the table, the index overhead is wasted and the search for data becomes a costly sequential process. For larger database systems, index information may become very large and ineffective. The index information cannot be compressed, because it must be readily available for a data request. Consequently, larger systems require a more novel means of handling and analyzing data, because of the reliance on index information.
Accordingly, there is a need in the art for a data management system for managing large volumes of data that is not dependent on index information.
There is also a need in the art for a data management system for managing large volumes of data that reduces the amount of necessary disk space required to store such data.
Additionally, there is a need in the art for a data management system for managing large volumes of data that reduces the amount of cost necessary to manage such data.